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Abstract. An artificial neural network can be used to generate a series of numbers. 
A boolean perceptron generates bit sequences with a periodic structure. The 
corresponding spectrum of cycle lengths is investigated analytically and numerically; 
it has similarities with properties of rational numbers. 



In the last fifteen years models and methods of statistical physics have successfully 
been used to understand emergent computation of neural networks. Several properties 
of infinitely large attractor and multilayer networks could be calculated analytically 
Such systems of simple units interacting by synaptic weights can be used as associative 
memory and classifiers; they are trained by a set of examples, detect unknown rules and 
structures in high-dimensional data, and store patterns in a distributed and content 
addressable way (Hertz et al 1991, Watkin et al 1993, Opper and Kinzel 1996). 

Another important application of neural networks is time series analysis (Weigand 
1993). But only recently statistical physics has been used to model training and 
prediction of bit sequences by a perceptron (Eisenstein et al 1995, Schroder et al 1996). 
A neural network is trained by a sequence of numbers; after the training phase the 
network makes predictions on the rest of the sequence. In analogy to generalization the 
training and test data are generated by a neural network, as well. It turns out that the 
generation of sequences of numbers by a perceptron or multilayer network is already 
an interesting problem which should be understood before prediction is investigated 
(Eisenstein et al 1995, Kanter et al 1995). 

This problem is a special case of the neuronic equations of Caianiello (1961). These 
equations, which were suggested to model a neuron including time dependency, can only 
be solved in special cases. Most work was done for one input and a memory back into 
time with couplings that decrease exponentially Wi = a 1 (a < 1). Several analytic results 
for the transients and the limit cycles of the resulting dynamics have been achieved for 
this case (for example Caianiello and Luca 1965 or Cosnard et al 1988). Only few work 
has been done for other weights (for example Cosnard et al 1988b and 1992). Our 
motivation for studying this recursion equation is to examine the generalization ability 
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and in a first step the ability of a perceptron to generate time series. Hence we have no 
restrictions on the weight vector a priori. 

Numerical analysis of a perceptron with random weights generating sequences of 
numbers shows that the sequences are related to the Fourier modes of the weight vector. 
Therefore it is useful to study weight vectors with a single mode only. In this case an 
analytic solution of a stationary sequence could be derived for large frequencies (Kanter 
et al 1995). 

This solution holds for continuous odd transfer functions, for example tanh(/fa). As 
a function of the slope (3 a phase transition to a nonzero sequence occurs. The phase 
of the weight vector results in a frequency shift of the attractor. In this letter we want 
to extend this solution to infinite slope (3 and general frequencies, that is we derive 
an analytic solution for the bit generator. We find a much richer structure of the bit 
sequences generated by a boolean perceptron compared to sequences of continuous ones. 

A bit generator (figure 1) is defined by the equation 

N 

S v = aga^w j S v - j {ue{N,N + l,N + 2,...}) (1) 

3=1 

where the S u G {+1, —1} is a bit of the sequence (So, Si, S2, S3, . . .) and w G M N is 
the weight vector of the perceptron of size N. As mentioned, it is useful to restrict the 
weights to one single mode. 

j 

Wj = cos(27rg — + 7T0) (2) 

where q G IV is the frequency and G [0, 1[ is the phase of the weights. Given an 
initial state (So, . . . , Sn-i) equation (1) defines a sequence (Sat, Sjv+i, • • •) which has to 
run into a periodic cycle of length L < 2 N . We try to find an analytic solution of the 
periodic attractor. 

Equation (1) may be expressed in terms of the local fields h v = jqY^^WjSu-f 

1 N ( j \ 
h » = Jj cos ( v 27r ^ ]y + sign^-j). (3) 

We have to solve this self-consistent equation for the function h u . Since simulations show 
that limit cycles are dominated by one frequency, we assume sign(/i„) is a periodically 
alternating step function with frequency k + r (with k G IV, r G [0, 1[), where the 
frequency is defined for the variable v/N: 

(k + t) 

sign(^) = sign(sin(27r — — — v)). (4) 

In the case that N is a multiple of 2(k + r), i.e. for integer wavelengths, we found an 
analytic solution of equation (1). With this ansatz the right hand side of equation (H) 
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is a periodic function with a period of length N/(k + r) and h v = —h u+N ^ 2 (k+r)) for 

— 11 fliir main r*=>Gnl+" ig 

2(fc+r) 



v E {0, . . . , nTTT-r — !}• Our main result is 



h 1 sin(i(T + l)(^ + 7 r)) v T nq nq, 

h v = t^tt : — r-r^r • sin [zirq \- d)7r -\ h 7rH 

iVsin(f) cos(^) 1 q N 9 2 { k + r } N' 

for T odd 

. , sin(07r - 22) 
+ — cos(0tt) + . ' v for T even 
iVsm(-$) 




(5) 



where we have abbreviated T = [(& + r)(2 — 2u/N)] using the Gaussian bracket [x] that 
denotes the closest integer less than x. For <fi = r = and k = q this results in 

2/c 2nku ixk 

K = NMfj sm( ^r + N ) (6) 

A sample function is plotted in figure 2. Note that it consists of two parts with the same 
frequency q. In the limit iV — > oo these parts are connected continuously. Equation (]3|) 
gives the condition 

h„>0 „ 6{0 ,...,^-l} (7) 

Figure 3 shows the possible frequencies that satisfy condition (0) within our ansatz. 
Only values of fc + r = iV/ (2i) with integers i are possible with our ansatz. 

For N — > oo a necessary condition for equation (^) is fro — 0. For k > q this 
is sufficient so the nontrivial ih v ^ 0) solutions are given by the values of r, k that 
fulfill the equation sin(07r + (2k + l)(7rg/ (2(/c + r)) + 7r/2)) = which is equivalent to 
g(2A; + 1) = (A; + r)(22; — 20—1) with an integer z. The frequencies k + r that are 
allowed from this condition are shown in figure 3, too. 

We see that the analytic solution of the sequence generator with a continuous 
transfer function (Kanter et al 1995) cannot just be extrapolated to the case of the 
bit generator. The continuous generator, close to the transition point and for k 3> 1, 
has k — q and r = <f), whereas we find a spectrum of solutions with k > q and r(q, (ft, k), 
as shown in figure 3. 

Up to now we have considered only integer wavelengths. Now we want to discuss 
the general case of arbitrary values of q and <fi. We want to address the two questions: 

(i) Do additional solutions consisting of one frequency exist? 

(ii) What are the properties of the bit sequences? 



We assume that there are solutions of the form (f|) with general frequencies k + r for 
a given system size N. As a function of q + we numerically scan the output frequency 
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k + t and determine the frequency of the limit cycle when the system was started with 
a sequence of frequency k + r. Some of these initial states stay at stable states with 
almost the same frequency. Other ones run to the lowest branch (k = q). Random 
initial states lead to the lowest branch with a very hight probability Figure 3 shows 
that the results of this simulation are in agreement with the extension of our equations 
(||) and (0) to general k + r which leads to allowed regions for for q + as a function 
of k + r. For the lowest branch k = q the phase <j) of the weights results in a frequency 
shift r of the bit sequence, with r « for q ^> 1 similar to the continuous case. 

The next problem is to understand the length L of the stationary cycle generated by 
the finite bit generator with one frequency in the couplings and a random initial vector. 
We consider the case q = k, only. Figure 5 shows the results of a numerical calculation 
of Equation (1) for N = 1024. Obviously L has a rich structure as the function of the 
phase (f> of the weights. For < (f> < | each cycle has only one maximum in the Fourier 
spectrum with frequency q + r(<f>). The numerical results show that in this case L is 
limited to the value 2N. Each value of L belongs to a whole interval in the 0-axis, 
but only to a single value of r. Hence, the function r(<j>) has a step like structure; r is 
locked at rational numbers as shown in Figure 6. The size of the steps decreases with 
increasing N and r. 

Can this structure of L(t) be understood from the extension of the analytic solution 
Equation @? For arbitrary values of r the sequence Si is quasi-periodic, in general with 
an infinite period L. However, if k + r is rational, k + r = r/s with integers r and s 
which are relatively prime, then the period L is given by the numerator of Ns/r. This 
means, that L is the smallest multiple of the wavelength N/ (k + r) that is an integer. In 
fact, in Figure 5 we have plotted all of these values of L for q + <p = r/s and L < 2N. For 
L < N all of these L values correspond to a cycle of the bit generator. For N < L < 2N 
the bit generator produces only even values of L whereas the analytic argument gives 
all integers L. 

For | < cf) < 1 the bit-generator essentially produces the same structure of L values 
as for < <p < |. However, there are always a few solutions which are mixtures of 
several modes k\ + T\ and ^2 + ^2; and which yield periods with L > 2N. For < r < | 
we never observed such mixtures of modes. 

As a side remark we notice that the structure of L(q + r) is essentially determined by 
properties of rational numbers, which might be discussed in high school mathematics. 
If the numerator p is plotted for each rational number x = p/r in the unit interval, we 
obtain Figure 7 (r < 800). Hence, above each rational number p/r a roof opens below 
which no other values of p appear. Each roof has the form l/\p — rx\. Low values of r 
have a wide roof. These results, which determine the structure of the cycle lengths of 
the bit generator, may be well known in number theory and nonlinear dynamics (circle 
map, winding number), but they have skipped our attention so far. 
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The upper bound of the cycle length L of 2N can be understood as follows. The 
analytic solution (|5]), extended to general values of k + t, yields quasi periodic bit 
sequences with infinite cycle lengths L = oo for irrational k + r. However, each 
cycle length has to be limited by the number of input strings for the deterministic 
bit generator, Equation ([!]), which gives L < 2 N . The last argument opens a 
different possibility to calculate L from equation (|5|). Let us start with the sequence 
(S , Si, ... , Sn-i) given by the equation (|5|), of the analytic solution. If a bit generator 
tries to follow this solution it can do so only if each input string (Si, Si +1 , . . . , Si + n-x) 
has not occurred before. Hence, the first appearance of a previous sequence 

(Si, SVfl, • • • , Si + n) = (Sl+L, Sl+L+l, • • • , Si + L+N-l) (8) 

defines a length L of a cycle. 

More insight can be achieved by examining the continued fraction expansion of 
2(k + t)/N: 

We define the expansion to order i to be S{ and U to be the denominators the s,: 

s = l = 1 

si = — k = a x 

S2 = t^t- h = axa 2 + 1 
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aiH — t 

a 2 + ~ 



^t— ^3 = 0ia 2 + l)a 3 + ai 




— ctih-i + ij— 2 

If 2(/c + t)/N is of the form Si = 1/ai the length L of the cycles as a function of iV 
can be given easily: 

for N < ax 

(11) 

for N >ax 

Is is obviously limited by 2N. 

For general 2(k + r)/N the continued fraction expansion reveals a hierarchy of of 
"defects", each having a period of /j, in the periodic structure of the resulting sequence. 
This leads to the the fact that for any length scale introduced by N , two identical 
subsequences of length iV with a distance less than 2N can be found. 

As a consequence the perceptron locks, for a given frequency k + r, in cycles that 
correspond to frequencies given by the continued fraction expansion of 2(k + t)/N, 
truncated at a certain depth. This explains the steps in figure 6. 
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Finally we point out similarities to the case of bit generators with exponentially 
decaying weights and additional bias (Cosnard et al 1988). In this case one finds cycles 
which are limited by N + 1. All of the cycles can be classified by rational numbers r/L 
where L is the length of the cycle, L < N + 1 and r is the number of positive bits in 
the cycle. 

In summary, we have obtained an analytic solution for the cycles of a bit-generator 
with periodic weight vectors. We found a whole spectrum of periodic attractors; the 
frequencies k + r depend in a complex way on the frequency q and phase of the weight 
vector of the perceptron. 

Numerical simulations showed that the bit sequences relax into cycles with lengths 
L, which are smaller than 2N. The structure of L as a function of k+r has been analyzed 
in terms of number theory. An analytic solution was given for certain frequencies; the 
extension to the measured frequencies results in a similar structure of cycles. 
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Figure 1. A perceptron learning a periodic time series. The desired output of the 
perceptron (marked) is the next bit of the series and therefore part of other input 
patterns as well. 





Figure 4. Possible solutions of the bit generator, frequency k + r of the solution 
versus frequency plus phase shift of the couplings q + <j>. Comparison of the simulation 
(x : N — 443) with the extrapolated analytic solution (lines). 
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Figure 5. Cycle length L of the BG as a function of the frequency q + <j> — rfs for 
N — 1024 (dots) and numerator of (Ns)/r (squares). Fbr k — 251, <f> G [0, 1[; only a 
part of the cycle lengths shown since the highest observed cycle length was 8000. 
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Figure 6. T as a function of <j> for N = 1024, k = q= 251 
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Figure 7. Numerator p of the reduced fraction x — pjr. (r < 



